Pattern recognition has been applied to a variety of problems, including handwriting recognition, speech recognition, image understanding, etc. A central problem of pattern recognition is the exploitation of known invariants in a pattern domain. Invariance represents a difference in individual target patterns (e.g., images) of a given classification that does not alter their recognized classification. For example, given two different instances of a handwritten numeral ‘9’ (see patterns 100 and 102 in FIG. 1), both instances are still recognized (or classified) as a numeral ‘9’. One approach to handling invariants is to introduce invariant information into the pattern classifier's training set, which is used to generate the classifier.
In images, known invariants may include rotation, translation, sheer, scaling, brightness and lighting direction. In addition, in handwriting recognition, invariants may include line thinning/thickening and other non-uniform character differences. Also, in speech recognition (e.g., phoneme classification), invariants may include differences in base frequency or duration. A wide variety of other invariants, singly or in combination, are also possible in a wide assortment of applications. When using machine learning approaches in these applications, it is a challenge to combine a training set with knowledge of invariants in order to obtain a satisfactory pattern classifier, particularly when the training set is small.
One existing method of incorporating invariants is to include discrete invariant samples in the training set. For example, one approach takes multiple samples of a handwritten numeral ‘9’ in the presence of invariants and adding them to the training set. Another approach generates discrete “virtual” invariant samples by the application of an invariance transformation and adds these virtual samples to the training set. However, these approaches can result in inconvenient or unwieldy training sets and, nevertheless, can result in incorrect classification caused by gaps between the discrete training samples.
Another method involves modeling invariants using a linear approximation. However, linear approximations of invariants are substantially inaccurate in most circumstances. Nevertheless, while non-linear approximations may provide greater accuracy, existing approaches have been unable to solve for non-linear approximations within given constraints.